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Abstract 

Introduction. This study investigated unmonitored referrals in a nationwide, 
collaborative chat reference service. Specifically, it examined the extent to 
which questions are referred, the types of questions that are more likely to 
be referred than others, and the level of user satisfaction with the referrals 
in the collaborative chat reference service. 

Method. The data analysed for this study were 420 chat reference 
transaction transcripts along with corresponding online survey 
questionnaires submitted by the service users. Both sets of data were 
collected from an electronic archive of a southeastern state public library 
system that has participated in 24/7 Reference of the Metropolitan 
Cooperative Library System (MCLS). 

Results. Referrals in the collaborative chat reference service comprised 
approximately 30% of the total transactions. Circulation-related questions 
were the most often referred among all question types, possibly because of 
the inability of 'outside' librarians to access patron accounts. Most 
importantly, user satisfaction with referrals was found to be significantly 
lower than that of completed answers. 

Conclusion. The findings of this study addressed the importance of 
distinguishing two types of referrals: the expert research referrals conducive 
to collaborative virtual reference sendees; and the re-directional local 
referrals that increase unnecessary question traffic, thereby being 
detrimental to effective use of collaborative reference. Continuing efforts to 
conceptualize referrals in multiple dimensions are anticipated to fully grasp 
complex phenomena underlying referrals. 

Introduction 

The collaborative virtual reference service is an innovative reference service that 
can bring many benefits to both service users and libraries. Thanks to 
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collaborations among libraries that share their staff, technology and other 
resources, no longer are library patrons confined to the services and resources of 
their local libraries. Patrons now have the opportunity to ask questions of 
institutions in remote locations for specialized information that would not 
otherwise be available to them. In many collaborative services such as 24/7 
Reference , patrons can access the service without being restricted by time and 
location. Thus, library patrons could obtain help for their information needs via 
the help of participating libraries in a reference consortium even when their local 
libraries are closed. 

From the library management perspective, libraries can deploy staff more 
efficiently by participating in a bigger pool and embrace otherwise more costly 
virtual reference services. Besides cost reduction, a member library of a 
consortium could answer difficult subject-based research questions more easily 
by referring the questions to subject specialists in other libraries in the 
consortium. However, it seems that a large proportion of questions asked in a 
collaborative service are simply referred back to the home libraries because they 
should be resolved at a local level f Coffman 2002 : Heifer 2003 1. Unfortunately, 
there is little monitoring of further transactions to ensure whether the patrons 
received satisfying answers. 

Unmonitored referrals, originally defined by Dewdney and Ross ( 1994 ), refers to 
a situation where the reference staff gives the patron a call number or refers the 
patron to a source in the library thought to contain the answer but does not follow 
up or check to make sure that the source is not only found but actually answers 
the question. These unmonitored referrals seem to be frequently observed in a 
collaborative virtual reference setting when 'outside' library staff refer the patrons 
back to local libraries. If a large proportion of those referrals are made back to the 
local libraries, however, that would not be an effective allocation of resources. 

More importantly, the referrals, if unattended, can easily frustrate the users of 
collaborative virtual reference services. While reference questions are often 
answered by outside library staff, many patrons would still view the service as a 
local service. Whenever the questions require local resources and services to 
answer adequately, library staff have to refer patrons back to their local libraries 
to complete the answer. Would the patrons still feel satisfied with a delayed 
answer? In fact, patrons' frustration caused by being transferred from one point to 
another in the midst of referrals is not unfamiliar. This frustration is equivalent to 
our everyday experience from using automated customer services on the phone or 
asking directions on the street. 

Although some studies have attempted to identify the types of questions that are 
effectively answered in virtual reference, few have examined the types of 
questions that are more prone to referrals, or user satisfaction with referrals in the 
context of collaborative virtual reference. Perhaps this is because referral are felt 
to be legitimate answers, although the notion has yet to be validated. Dewdney 
and Ross (1994) pointed out the problem of a high proportion of unmonitored 
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referrals (42%) at the physical reference service desk. They maintained that 
reference staff tend to make unmonitored referrals as a way of circumventing the 
reference interview. Because few studies have examined the unmonitored 
referrals in collaborative virtual reference service settings, the present study 
investigates the extent of referrals and user satisfaction with referrals in these 
types of services. Specifically, the current study focuses on the following three 
questions: 

Research Question 1: To what extent are questions answered as referrals and to 
what extent are questions answered completely during collaborative virtual 
reference sessions? 

Research Question 2: Are certain types of questions referred more frequently than 
others? If so, what types of questions are they? 

Research Question 3: Are there any differences in user satisfaction levels 
betweentransactions answered with referrals and transactions answered with 
completed answers? 

The findings of the study will inform us about the question types that are 
frequently referred in the collaborative virtual reference service. The results will 
also inform of the extent to which referrals are made in collaborative virtual 
reference and the efficacy of the referrals as an acceptable answer category from 
the users' perspective. It will also help local libraries design their services for 
optimal question traffic management in the collaborative service. 

Literature review 

Collaborative virtual reference services 

Virtual reference services have become a critical component in current reference 
practice where the intent of libraries is to provide a fully functioning user service 
in the digital library environment. The rapid growth in the service has been made 
through both local and national efforts ( Kresh 2002 ). Some critics question the 
cost-effectiveness of virtual reference services because of low usage and high cost 
( Coffman 2002 : Coffman & Arret 2004 ). Based on a cost-benefit analysis, 
Coffman and Arret suggest the costs associated with software, training and 
staffing are 'a pretty expensive way to answer a question' and may not justify 
continuing the service. 

Others argue that virtual reference is a niche service that will merge into a part of 
reference services just as happened to telephone reference services in the 1930s 
( Kern 2004 ). In particular, responding to the negative view toward virtual 
references, many suggest joining a consortium as a way for those libraries that 
cannot afford a stand-alone virtual reference service ( Bailev-Hainer 2005 : Kresh 
2002 : Tenopir 2004 T Examples of collaborative models include an international 
library network project, the Collaborative Digital Reference Service (CDRS), the 
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CLEVNET Library Consortium of Ohio public libraries, and AskColorado . 
Because of many benefits that collaborative service brings to both library 
management and user services, it is estimated that approximately seventy-six 
collaborative virtual reference services were in operation through the active 
participation of libraries of various types and sizes as of November 2004 ( Sloan 
2004 1. 

Referrals in collaborative virtual references 

Reference staff seem to make referrals for their patrons for various reasons. The 
cases can be divided roughly into two types: expert research and re-directional 
local referrals. Expert research referrals relate to the situations when the questions 
are referred to subject specialists who could answer with their high level of 
expertise and resources. Difficult research questions would fall into this type. The 
referrals in the other type, re-directional local referrals, relate to the situations 
when the answers require specific local knowledge or locally-restricted databases. 
Questions that would fall into this type include questions about local library 
events that may or may not be listed on the library's Website or patrons' requests 
to check overdue fines. Despite both being referral, each type of referral could 
have considerably different implications for reference service delivery. The 
former type of referral enables member libraries to coordinate human expertise 
and resources in order to answer the questions that are otherwise unanswerable. In 
this case, patrons are linked to richer resources and subject specialists. 

In contrast, re-directional local referrals can be ineffective or even result in a 
disservice to the patrons. That is, patrons who access the inter-institutional 
collaborative network may not necessarily understand the seamless nature of the 
service that goes beyond the walls of their local library. It is possible that people 
who access this service through their library portal would ask simple, circulation- 
related questions (e.g., overdue status), anticipating an immediate answer. 
However, many times, these questions cannot be completed by outside staff who 
do not have authority to access patron records. In this case, the patron should be 
referred back to her or his local library to get the final answer. If a high 
proportion of the questions submitted to collaborative services are referred to a 
local library for this latter reason, it becomes an ineffective deployment of 
resources for the local library. Having this concern, some libraries may decide to 
remain local for their virtual reference service, understanding that a high 
proportion of their reference questions relate to their library resources and 
services ( Kibbee et al. 2002 ). 

In the current literature, there are not many empirical studies that have examined 
the proportion of referrals in reference services. Among the few, Ross and Nilsen 
( 2000 ). in their unobtrusive reference service evaluation study, reported that 37% 
of the total reference transactions were unmonitored referrals. This finding is 
based on 261 reference transactions that occurred at physical reference desks in 
both public and academic library settings. 
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As libraries adopt collaborative virtual reference services, librarians tend to 
expect more research related referrals because collaboration and communication 
among institutions have become much easier and seamless in the online network. 
Interestingly, a rather reverse finding was reported in a recent study by Nilsen 
( 2004 ) who compared the unmonitored referrals between physical and virtual 
reference settings. While more than one-third of the transactions have 
unmonitored referrals in the physical setting, the proportion was somewhat lower 
in the virtual setting, comprising 28% of a total of 40 transactions. In her updated 
study with more transactions, unmonitored referrals were reported as 38% of a 
total of 85 virtual reference transactions where chat reference comprised 25 
transactions ( Nilsen 2005 ). Since this small sample-size research appears to be the 
only one that investigated referrals in virtual reference setting, referrals in 
collaborative virtual reference are still largely unknown. 

Referrals by query type 

Are there any specific types of questions that result in more referrals than others 
in the collaborative virtual reference setting? In other words, would referrals be 
contingent on the types of questions? Indeed, most factual and subject-based 
research questions are expected to be answered during the session by most 
participating librarians, especially when the consortium is equipped with a shared 
online catalogue and consortia-shared databases. Among these questions, 
complex subject-based reference questions would be more likely to be referred to 
subject experts within the consortium. However, this type of question was 
reported to be very infrequent among the transactions. For example, Lee ( 2004 1 
reported that research level questions comprised only 3% of all transactions in a 
virtual reference environment at an Australian academic library. 

A previous study suggests another type of question that would result in high 
referral rates. According to Kibbee et al. ( 2002 ). a service that generates a high 
proportion of local questions argues against the inter-institutional collaborative 
service because the questions would not be effectively answered by outside 
librarians in the consortium. This is because questions that involve local 
knowledge (e.g., history, events, or services) or questions that are answerable 
only by local staff (e.g., use of locally restricted databases or access to patron 
accounts) tend to result in more referrals back to local libraries. Yet, these 
speculations should be empirically tested to identify the types of questions that 
produce more referrals in a collaborative service. The results will serve as 
important managerial information for designing the practice and policies for 
virtual reference services. 

User satisfaction with referrals 

While the professional community generally encourages the use of referrals as an 
alternative to actual answers ( Reference and User Services Association 2004 1. 
Dewdney and Ross (1994) revealed a pitfall of the referral practice by reporting 
its ineffectiveness from the users' perspective. In fact, this contention is supported 
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by a collaborative online chat reference use study ( Kwon 2004 1. According to the 
finding, a mere suggestion of alternative resources or making referrals was not a 
positive predictor of user satisfaction. That is, unmonitored referrals do not 
warrant user satisfaction because the question might or might not be ultimately 
answered. 

A handful of studies has investigated the impact of referrals on reference 
effectiveness in traditional reference settings. In the study by Dewdney and Ross 
(1994), most patrons who received unmonitored referrals reported that they found 
nothing or very little helpful material when they had to search the sources to 
which reference staff referred them. In fact, Douglas 1 1988 ) categorized these 
referrals as one of the six types of reference service failures. Furthermore, Murfin 
and Bunge ( 1984 1 reported that the average reference success rate dropped from 
69% to 22% when the librarian was busy and, as a result, made suggestions that 
were not followed up, instead of actually helping the patron with the search. 
Murfin and Bunge maintained that the practice, policy or necessity of directing 
the user rather than accompanying the user on the search was a factor that caused 
librarians to perform far below their potential. All of the above studies indicate 
apparent problems in the effectiveness of referrals in the traditional physical 
reference setting. Yet, there is still a question of how patrons would feel when 
their questions were referred as opposed to when their questions were answered 
completely. Furthermore, it is also a question how patrons would feel about 
referrals in a collaborative virtual reference setting. 

While many studies have examined the user satisfaction with virtual reference in 
general, few researchers have investigated the satisfaction of the patrons whose 
questions are answered as referrals in the collaborative virtual reference setting. 
The present study investigated this issue empirically. 

Method 

Setting and participants 

The present study examined chat reference services delivered through the 
Broward County public library system in Florida. As the largest library system in 
Florida, it has 33 regional and branch libraries. Since August 2002, the system has 
used the 24/7 Reference chat reference service delivered by the Metropolitan 
Cooperative Library System (MCLS), an association of libraries located in the 
greater Los Angeles area, funded by a Federal Library Services and Technology 
Act (LSTA) grant. 

The data analysed for the present study were on-line chat reference transactions 
initiated by the patrons of the Broward Country library system, along with online 
survey questionnaires, which the service users completed voluntarily. The 
questionnaire was devised and supplied by 24/7 Reference and popped up upon 
completion of each transaction. While the patrons were mostly the users of the 
library system, the reference staff who provided the service were from forty-nine 


http://www.infomiationr.net/ir/l l-2/paper246.html[6/21/2016 3:36:16 PM] 






User satisfaction with referrals at a collaborative virtual reference service Virtual reference services. Reference services, Referrals, User satisfaction 


library systems across the United States participating in the MCLS 24/7 
Reference programme. During the six-month research period between January 
and June 2004, a total of 1,667 transactions took place and 1,387 were analysable 
as being either completed or transferred transactions. In order to analyse the 
influence of referrals on user satisfaction, all 420 transactions that had a 
corresponding questionnaire that chat service users had submitted immediately 
after their session to report their satisfaction with the service were chosen. This 
amounted to 30.1% of the total analysable transactions. By having a chat 
reference session transcript with a matching user satisfaction survey 
questionnaire, the researcher was able to examine what question was asked in a 
reference session, whether the question was answered completely or referred, and 
to what extent the patron was satisfied with the answer or referrals she or he 
received. 

Measurement of research variables 

User satisfaction 

User satisfaction was assessed by three indicators: satisfaction with the answer, 
perceived staff quality, and willingness to use the service again. These indicators 
were measured by patrons' responses to three questionnaire items. First, 
satisfaction with the answer was measured by asking the question, 'Were you 
satisfied with the answer you received to your reference question?' Respondents 
answered the question by choosing one from 'Yes', 'Not sure' and 'No'. The result 
shows that, among the 417 respondents who answered this question, 65.2% 
reported that they were satisfied with the answer received, and 21.1% of the 
respondents were not sure whether they were satisfied or not. Finally, 12.6% of 
the respondents reported that they were not satisfied at all. 

Perceived staff quality, the second indicator of user satisfaction, was measured by 
the user response to the question, 'The quality of the library staff service in 

answering this request was _'. The answer choices were 'Excellent', 

'Good', 'Average' and 'Poor'. According to the finding (N = 416), 68.2% of the 
respondents answered that the librarians handling the reference questions were 
'Excellent'. About 19.5% answered 'Good' and 11.3% answered 'Poor.' 

Finally, Willingness to use the service again, the third indicator of user 
satisfaction, was measured using the questionnaire item, 'Will you use this service 
again?' The answer choices were 'Very likely', 'Maybe' and 'Never'. 77.2% of 
users answered that they were very likely to use the service again 19.0% 
answered maybe, and only 3.8% said they would never use the service again (N = 
417). 

The overall user satisfaction was computed by summing up the above three 
indicators ( Footnote 1 ). Measurement research literature indicates that composite 
variables yield scores that are generally more valid and reliable than does a single 
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item (DeVellis 1991). 

In order to determine the reliability and validity of the scores yielded by the 
composite variable, a reliability test and a factor analysis were conducted, 
respectively. As a result, a Cronbach's alpha coefficient of .85 was obtained, 
indicating that the composite variable generated scores were reliable. The factor 
analysis revealed that the three items were represented by one factor, with 
structure-pattern coefficients of .868, .916, and .876. This factor explained 
78.64% of the total variance. This high validity score confirms that the composite 
variable is measuring a single construct, suggesting that the composite variable is 
a valid measure of user satisfaction. Finally, the mean of the overall user 
satisfaction was 12.69 with a standard deviation of 3.44 in the range between a 
maximum value of 3 for 'highly dissatisfied' and the minimum value of 15 for 
'highly satisfied.' 


Answer completeness 

Answer completeness refers to the level of completeness that library staff answers 
to the patron's inquiry during the reference transaction. A content analysis of 420 
online chat reference transaction transcripts was conducted in order to code the 
levels of answer completeness. A total of 16 different levels of answer 
completeness were identified from the initial coding (See the first three columns 
of Table 1 ). 

Later, the initial sixteen levels of answer completeness were collapsed into four 
broader categories: completed answer, partial or no answer, referrals, and 
problematic ending (See the last three columns of Table 1 ). This reduction in the 
categories helped to remove some ambiguities in the micro-level coding. For 
example, when a transcript showed an abrupt ending, it was not clear whether it 
was patron's termination due to delay, librarian's premature ending, or 
technological failure. Thus, a broader category of 'problematic ending' 
incoiporates all of these possibilities constituting an abrupt ending. A fewer 
number of answer categories also enables bivariate data analysis that is necessary 
to answer the research questions. Thus, the final coding for answer completeness 
categories were defined as follows: 

• A transaction was coded as 'completed answer’ if a transcript indicated that the 
librarian offered the patron direct answers or sources that may include answers to the 
patron's question and that the librarian ends the transaction with a proper closure (i.e., 
Librarians asks the patron if the question have been completely answered and the 
patron responds positively.). Additional sources of information may be provided to 
the patron along with the answer to the initial question. In particular, in order to 
determine a proper closure, specific behavioral items that are prescribed in the 
'Guidelines for Behavioral Performance of Reference and Information Services 
Providers' 1 2004 1 were employed. It is the Guidelines that the Reference and User 
Services Association (RUSA) developed to train reference staff and to assess their 
performance. The following four items were specifically selected from the 
Guidelines: 
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o 4.7: Asks the patrons if additional information is needed after an 
initial result is found (e.g., 'is there anything else I can help you 
with?') 

o 5.1: Asks patrons if their questions have been completely 
answered (e.g., 'did you find what you needed?,"does this 
completely answer your question?') 
o 5.2: Encourages the patrons to return if they have further 
questions (e.g., 'If you don't find what you are looking for, 
please come back and we'll try something else.') 
o 5.9: Takes care not to end the reference interview prematurely. 

• A transaction was coded as 'partial or no answer’ if a transcript indicated that the 
patron's initial question was either partially answered or unanswered and no further 
reference to other sources or services was provided. 

• A transaction was coded as 'referrals' when a transcript indicated that the patron's 
initial question was either partially answered or unanswered during the reference 
transaction but reference staff gave the patron information sources or contact 
information that may or may not contain the answer. The reference staff did not check 
if the sources had answered the patron's question. 

• A transaction was coded as 'problematic ending' when a transaction was ended before 
the patron received the answer due to disconnection, delayed answer, or without clear 
reason. It also includes librarians' premature ending without proper closing remarks 
and system failure due to connection problems. 

Question types 

A content analysis of transaction transcripts was undertaken to identify the types 
of questions that are submitted to the collaborative virtual reference. A total of 
fourteen categories of question types emerged initially. Again, the fourteen 
categories were collapsed into five broader question types: simple, factual 
questions, subject-based research questions, resource access questions, 
circulation-related questions, and local library information inquiries. Below list 
the definitions of the five types of questions with examples: 

• Simple, factual question refers to known-item search questions, directional questions, 
or a question of a factual nature that can be answered quickly by consulting only one 
or two reference tools t Gamsey & Powell 2000 T Examples are 'Is the book Funny 
Laws and Other Zany Stuff by Sheryl Lindsell-Roberts available?', 'What does 
'borrower status: restricted’ mean?', and 'What is the correct way to write this name, in 
last name first order: Dr. James R. J. Smith, III?' 

• Subject-based research question refers to a reference question requesting a particular 
kind and number of books or journal/magazine articles on a specific topic, teaching 
the steps of the research process, locating hard to find literature, and guidance with 
database choices I Marsteller & Mizzv 2003 : Ward 2004 1. Examples are 'Can you tell 
me how to celebrate children's day in America?', 'I want to know the titles of 5 books 
that deal with the development of gender identity', and 'Where can I find a current 
article on the culture of any indigenous tribe?' 

• Resource access question refers to an inquiry about access and use of library 
catalogues, databases or other library resources. Examples are 'How do I access the 
Oxford English Dictionary through the library Website?', 'I'm having trouble 
accessing my library account. Can you help?' and 'Where can I access NetLibrary to 
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read an e-book?' 

• Circulation-related question, for the purposes of this study, refers to an inquiry about 
circulation policies (e.g., overdues, fines, renewal), online renewals, patron account 
checking, or other circulation related issues. The answer may or may not be readily 
found from the Website of the local library. Questions that fall in this category 
include 'Can you check which books are overdue in my record?', and 'I saw I have a 
60 dollar fine. Is there a payment plan or forgiveness policy I could do to be able to 
check out books again?' 

• Local library information inquiries, for the purposes of this study, refers to a question 
that involves information about a local library. Questions that fall in this category 
include 'Can I use a typewriter in Sunrise branch library?' 'I am looking for a job at 
this library', and 'Where should I e-mail comments about Thursday's 'Balancing the 
Books' seminar at the main library? There was no box into which we could 
anonymously drop the evaluation forms’. 

Inter-coder reliability 

Inter-coder reliability tests were undertaken to ensure the consistency of coding 
for both question types and answer completeness. The tests show the level of 
agreement between two independent coders. To test the inter-coder reliability, the 
primary researcher coded the entire 420 transactions. Subsequently, the second 
coder, a reference librarian who received training for coding the RUSA guideline 
categories, coded every fifth transaction of the 420 transactions {n = 84). This 
number comprised 20% of the total transactions, which is a recommended 
percentage for social science research ( Neuendorf 2002 ). A widely used Cohen's 
Kappa (k) was used as the inter-coder reliability index for this study and was 
calculated using SPSS 13.0 ( SPSS Inc. 2004 ). Cohen's Kappa coefficient was 
found to be 1.00 for question type and .75 for answer completeness, indicating 
high agreements between the two coders in classifying both variables. 

Results 

Proportion of referrals and answer completeness 

Research Question 1 examines 'To what extent are questions answered as referrals 
and to what extent are questions completely answered during collaborative virtual 
reference sessions?' Both Table 1 and Figure 1 show the results of answer 
completeness based on total transactions (N = 420). 


Initial coding 
category 

Freq. 

Percent 


Final 

coding 

Freq. 

Percent 

Answered 

completely 

204 

48.6 


Completed 

answers 



Answered & 
transferred 

12 

2.9 


237 

56.4 

Answered & referred 

21 

5.0 





Ended with partial 
answer 

12 

2.9 


Partial or 
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Unanswered or 
failed 

8 

1.9 

Part answered & 
transferred 

4 

1.0 

Part answered & 
referred 

11 

2.6 

Unanswered & 
transferred 

6 

1.4 

Unanswered & 
referred 

5 

1.2 

Referred or 
suspended 

84 

20.0 

Transferred 

12 

2.9 

Patron disconnected 
before answered 

8 

1.9 

Uncertain ending 

1 

0.2 

Disconnect due to 
delay 

19 

4.5 

Librarian's 
premature ending 

7 

1.7 

System failure 

6 

1.4 

Total 

420 

100.0 


no answer 

20 

4.8 

Referrals 

122 

29.0 

Problematic 

41 

9.8 

ending 

Total 

420 

100.0 


Table 1: Content analysis of answer completeness 

These findings revealed the answer to Research Question 1; referrals reached up 
to approximately 30% of all transactions in the collaborative virtual reference 
service. In contrast to referrals, completely answered transactions comprised 
56.4% of the transactions, which is very close to the well-known 55% reference 
success rate ( Hernon & McClure 1986 : Saxton & Richardson 2002 ) 

Question types 

A profile of questions that public library patrons asked of collaborative virtual 
reference was found from a content analysis of transaction transcripts. Figure 1 
presents the findings (N= 415). 


http://www.infomiationr.net/ir/l l-2/paper246.html[6/21/2016 3:36:16 PM] 



























User satisfaction with referrals at a collaborative virtual reference service Virtual reference services. Reference services, Referrals, User satisfaction 


Local library 
information, 6.80% 


Circulation-related, 

48.90% 



Simple factual 
questions, 9.60% 


Subject-based 
research, 25.80% 


Resource access, 

8.90% 

Figure 1: Proportions of five question types (N = 415) 

(*This analysis was based on 415 transactions after eliminating five ambiguous transcripts.) 


As shown in Figure 1, circulation-related questions were the most frequently 
asked questions of all (48.9%), followed by subject-based research questions 
(25.8%), simple factual questions (9.6%), resource access (8.9%) and local 
library-related information inquiries (6.8%). It should be noted that the high 
proportion of circulation-related questions seem to partly attribute to the fact that 
the circulation page has a link to the service. 


Relationship between question types and answer completeness 

Research Question 2 relates to whether certain types of questions are more 
frequently referred than others. If there are, what types of questions would they 
be? Figure 2 shows the degree of answer completeness by question types. 


□ Completed Answers □ Partial/No answers □ Referrals □Problematic Ending 

0 s 


a> 90 



Simple Factual Subject-based Resource Circulation- Local Library 
Question Research Q. Access Q. related Q. Information Q. 


Figure 2: Answer completeness by question types (N = 415) 
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Figure 2 demonstrates that both simple factual and subject-based research 
questions were completely answered over 70% of the time and their average 
referral rates were 5% and 11.2% of total transactions, respectively. In the case of 
circulation and local library information inquiries, answer completion rates were 
approximately 50% and referral rates were 44.8% and 39.3%, respectively. 

To determine a systematic association between answer completeness and question 
types, a chi-squared test was conducted using a significance level of .05. 
According to the test result, there was a statistically significant difference in 
answer completion rates across the five question types, ? 2 = 71.616 (12 d.f, A = 
415), p < .001. The effect size ( Footnote 2 ) associated with this relationship, as 
measured by Cramer's V, was .24, which suggests a small to medium effect size. 
This result indicates that the degree of answer completion is highly correlated 
with the types of questions. Specifically, this result shows that referrals occurred 
far more frequently in circulation and local library information inquiries while 
substantially fewer referrals were made in both simple factual and subject-based 
research questions. 

User satisfaction with referrals 

Research question 3 relates to whether there is any difference in the user 
satisfaction between referrals and completed answers? To answer this question, 
the level of user satisfaction was compared across the four answer categories. 
User satisfaction was highest when the questions were completely answered 
(Mean = 14.21; SD = 1.88; n = 237), followed by referred (Mean = 11.83; SD = 
3.40; n = 122), partial/unanswered (Mean = 10.68; SD = 3.91; n = 20), and 
problematic endings (Mean= 7.54; SD = 3.87; n = 41). With these results, an 
analysis of variance (ANOVA) test was conducted to determine if the user 
satisfaction of referrals is statistically significantly lower than that of completed 
answers (Figure 3) 
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Figure 3: User satisfaction by answer completeness (N = 420) 

The ANOVA test results revealed a statistically significant difference in user 
satisfaction across the four answer categories at the significance level of .05, F(3, 
416) = 80.341, p < .001. A post-hoc test of the overall ANOVA test was 
conducted to determine how the four groups differ by employing Games-Howell 
test. The Games-Howell test is considered to be robust when sample sizes and 
variances are not equal across compared groups ( Field 2005 : SPSS Inc. 2004 T 
The test result revealed that user satisfaction was rated into three different levels, 
as shown in Table 2. 


Answer 

completeness 

User satisfaction 


High 

Middle 

Low 

Completed 

answers 

14.21 (1.88)* 



Referrals 


11.83 (3.40) 


Partial or no 

answer 


10.68 (3.91) 


Problematic 

Endings 



7.53 (3.87) 


*The numbers in parentheses are standard deviations. 


Table 2: Post hoc ANOVA test using the Games-Howell test 

As Table 2 shows, patrons who obtained complete answers were most satisfied 
among all groups. Patrons who received referrals were significantly less satisfied 
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than the patrons whose questions were completely answered; however, patrons 
who received referrals fell into the same satisfaction level as those whose 
questions were partially answered or unanswered. More importantly, the current 
study revealed that users were far less satisfied with the referrals than with 
completed answers and experience only about the same degree of satisfaction as 
users receiving either a partial answer or no answer at all. Finally, all three groups 
were statistically significantly more satisfied than patrons who experienced a 
problematic ending. 

Discussion 

This study investigated referrals in a nationwide, collaborative virtual reference 
service. Specifically, it examined the extent to which questions are referred in the 
collaborative virtual reference environment, the types of questions that are more 
likely to be referred, and whether patrons would be equally satisfied with the 
referred answers as with complete answers. This study employed both content 
analysis of the transcripts of 420 virtual reference transactions that occurred 
between January and June 2004 and corresponding self-reported survey 
questionnaires that were submitted by the patrons after each transaction. 

According to the results of this study, approximately 30% of the questions were 
answered with referrals. Circulation-related questions were more frequently 
referred back to the local libraries than simple factual or subject-based research 
questions, mostly due to restriction of access to patron records. The latter two 
types of questions are non-local, generic questions in nature. These types of 
questions tend to be more completely answered than either circulation or local 
library information inquiries which involve more locality-specific knowledge. 
These findings indicate that questions involving local resources tend to create 
more referrals than questions that involve generic knowledge. More importantly, 
the current study revealed that users were far less satisfied with the referrals than 
with completed answers and experience only about the same degree of 
satisfaction as users receiving either a partial answer or no answer at all. 

Regarding this problem of lower user satisfaction with referrals, libraries 
participating in collaborative services could consider several approaches for 
handling this problem. An immediate measure would be to specify the scope of 
questions explicitly and effectively on the virtual reference service Web site. As 
another approach, Braxton and Brunsdale ( 2004 1 suggested that virtual reference 
services ask users to check their question types and to describe their questions 
using a Web form where users can also read the scope of questions available from 
the service. 

Probably a more effective measure to correct the phenomenon of problematic 
referrals would be to place the links to the virtual reference service in the most 
natural and pertinent locations, that is, at the point of need. A problem that seems 
to have contributed to the high volume of circulation-related questions in this 
study is the location of the link placed on a circulation page. Whenever patrons 


http://www.infomiationr.net/ir/l l-2/paper246.html[6/21/2016 3:36:16 PM] 



User satisfaction with referrals at a collaborative virtual reference service Virtual reference services. Reference services, Referrals, User satisfaction 


encountered a circulation-related problem, they seemed to click on the link to get 
help. However, a large proportion of these circulation-related questions were 
answerable only by authorized personnel in the patrons' home libraries. 

This finding supports Wells's ( 2003 ) argument that proper location of the links to 
the virtual reference services can enhance service effectiveness. According to 
Gray ( 2000 1. both the placement of links to the services on individual Web pages 
and the number of links to the service throughout the Web site directly affect the 
amount of question traffic. All these findings in the literature suggest a possible 
influence of link locations on the types of questions asked of virtual reference. 

However, the results of this study do not support Gray's other recommendation, 
'services that provide links on the bottom of all pages, on a main navigation bar, 
or on a help menu featured throughout the site will provide users with more 
opportunity to ask questions at the point of need' ( Gray 2000: 369 1. According to 
the findings of this study, not all links placed at the point of need helped the 
patrons especially when it does not guarantee answers. For instance, the present 
study reported 45% of questions asked via a link on the circulation page were 
answered as referrals and therefore patrons had to contact their home library 
again. These findings also suggest that a link on the circulation page 
unnecessarily increases circulation question traffic within the collaborative 
service. As the current findings indicate, user satisfaction with referrals was 
significantly lower than completed answers. Patrons seemed to have experienced 
confusion and frustration or felt they had wasted their time in this process. 
Referrals made back to the local library would work against the anticipated 
benefits of effective allocation of resources. Regarding this finding, a usability 
study could be conducted to empirically examine relationships between the 
locations of links to virtual reference and the service effectiveness. 

Despite the apparent problems in some referrals, they seem to be generally 
encouraged and reference staff seem to believe referrals are legitimate answers to 
reference questions ( Reference and User Services Association. 2004 ). However, 
the fact that librarians are advised to offer referrals when they do not provide the 
answer does not necessarily mean that referrals are effective in all circumstances. 
The findings of this study suggest that librarians should be able to distinguish 
between two types of referrals: the 'expert research' referrals conducive to 
collaborative virtual reference services (e.g., sharing expertise and resources in 
answering difficult subject-based research questions); and the 're-directional, 
local' referrals that increase unnecessary question traffic, thereby being 
detrimental to effective use of collaborative reference (e.g., referring patrons back 
to local libraries to resolve circulation disputes). A practical implication of this 
finding is that two types of referrals should be properly addressed in reference 
staff training so that staff can make referrals most effectively. A future study 
could compare the two types of referrals to further understand contingent 
conditions of those referrals. In relation to this, another interesting agenda of 
further research would be to compare the patrons and the reference staff for their 
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perceptions of and attitudes toward referrals. 

Conclusion and implications of the study 

While there are many expectations and promises in the emerging collaborative 
virtual reference service, the present study demonstrated some gaps in the current 
service delivery by investigating referrals. One limitation of the present study, 
however, is the fact that the study was conducted at a single public library system 
although the librarians investigated were from forty-nine different library systems 
participating in a nationwide collaborative chat reference programme. Thus, the 
findings of the present study should be further confirmed by replicating the study 
in libraries of different types and sizes. 

As another limitation, it should be noted that the present study included only the 
transactions that had a completed questionnaire. These transactions comprise only 
30% of the total of 1387 analysable transactions. Considering the possibility that 
the patrons who completed the questionnaire may not represent the entire user 
population accurately, the findings of the present study should be substantiated by 
further research. 

Despite these limitations, the results of this study provide several implications for 
future collaborative virtual reference practice and research. First, from the 
practical perspective, the current study empirically demonstrated that generic 
reference questions, such as simple factual and subject-based research questions, 
are more effectively answered than locality-specific questions. These findings can 
help local libraries make informed decisions on such issues as whether they go 
global or remain local in providing the service. The study also demonstrated 
empirically that the patrons did not value referrals to a great extent. This finding 
suggests that librarians should not regard all referrals as equally good alternatives 
to completed answers. Most importantly, local libraries can minimize confusion 
and frustration of their users by evaluating the current locations of links to virtual 
reference and manage the question traffic to provide proper directions for the 
patrons. This should also reduce the frustration of reference staff who might have 
to end their sessions without offering satisfying answers and help them to spend 
their time and expertise more wisely with the questions that they could better 
answer. 

From the reference research perspective, it can be contended that the present 
study addressed complex phenomena underlying a seemingly simple behaviour of 
making referrals. The findings certainly redirect our attention to the concept of 
'unmonitored referrals' that Dewdney and Ross ( 1994 ) had proposed from their 
research in the physical reference setting. While supporting their claims about the 
problems in unmonitored referrals, the present study suggests that referrals could 
bring both positive and negative implications to collaborative virtual reference 
services. In this regard, the use of referrals must be one such area that Whitlatch 
( 2004 ) pointed out to assure patrons' satisfaction in the current reference service 
environment where new technology can create both opportunities and threats. 
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Since further understanding of referrals is critical to maximize the user 
satisfaction in collaborative services, continuing efforts to conceptualize referrals 
in multiple dimensions are anticipated to fully grasp complex phenomena 
underlying referrals. 

Notes 

1. In creating a composite variable, a series of computations was conducted 
because the three items in the survey questionnaire were measured on a three- 
point or four-point ordinal scale (e.g., 'satisfied,' 'not sure,' and 'not satisfied' for 
the 'satisfaction with the answer' item). These ordinal level measures are not 
suitable to undertake inferential statistical tests, such as ANOVA, that are 
necessary to answer research question 3 of the present study. To resolve this 
conflict, the existing three questionnaire items were recoded and transformed into 
a composite variable to increase the variability of the measure. First, the ordinal 
level data variables were rescaled on a five-point scale as shown below: 



Negative 




Positive 


1 

2 

3 

4 

5 

Satisfaction with the answer 

Unsatisfied 

Not sure 



Satisfied 

Perceived staff quality 

Poor 


Average 

Good 

Excellent 

Willingness to use the 
service again 

Never 


Maybe 


Very likely 


Adjusted five-point scale 

As shown on the above diagram, for satisfaction with the answer item, 1 was 
assigned for 'unsatisfied,' 2 for 'not sure,' and 5 for 'satisfied.' For perceived staff 
quality item, 1 was assigned for 'poor,' 3 for 'average,' 4 for 'excellent.' For 
willingness to return item, 1 was assigned for 'never,' 3 for 'maybe,' and 5 for 
'very likely.' 

After the three items were rescaled on the same five-point scale, they were 
summed as a single composite variable, with a minimum value of 3 and a 
maximum value of 15. This procedure of data management allowed the researcher 
to conduct the necessary inferential statistical tests. 


2. The 'effect size' measured the degree of the strength of the association between 
the two categorical variables. Many behavioral science research papers now 
report effect size along with significance test result because of the limitations of 
the significance test. Effect size is useful information especially if the significance 
test result is sensitive to the sample size, as with Chi-squared tests. For this paper, 
I measured the effect size using Cramer's V which is a measure of the effect size 
of the Chi-squared tests f Klinc 2004 1. 
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